On the Circuit Complexity of Neural Networks

K. Y. Sill Information Systems Laboratory Stanford University Stanford, CA, 94305 T. Kailath Informat.ion Systems Laboratory Stanford U ni versity Stanford, CA, 94305 '~le introduce a geometric approach for investigating the power of threshold circuits. Viewing n-variable boolean functions as vectors in 'R'2", we invoke tools from linear algebra and linear programming to derive new results on the realizability of boolean functions using threshold gat.es. Using this approach, one can obtain: (1) upper-bounds on the number of spurious memories in HopfielJ networks, and on the number of functions implementable by a depth-d threshold circuit; (2) a lower bound on the number of ort.hogonal input. functions required to implement. a threshold function; (3) a necessary condit.ion for an arbit.rary set of input. functions to implement a threshold function; (4) a lower bound on the error introduced in approximating boolean functions using sparse polynomials; (5) a limit on the effectiveness of the only known lower-bound technique (based on computing correlations among boolean functions) for the depth of threshold circuit.s implement.ing boolean functions, and (6) a constructive proof that every boolean function f of n input variables is a threshold function of polynomially many input functions, none of which is significantly correlated with f. Some of these results lead t.o genera.lizations of key results concerning threshold circuit complexity, particularly t.hose that are based on the so-called spectral or Ha.rmonic analysis approach. Moreover, our geometric approach yields simple proofs, based on elementary results from linear algebra, for many of these earlier results.